Frequentist vs. Bayesian Approach

• The Frequentist Approach
  • Typical linear model

Y = \beta_0 + \beta_1X + \varepsilon

• Y : Dependent variable, the outcome
• \beta_0 : y intercept
• \beta_1 : The regression coefficient
• X : Independent variable
• \varepsilon : Random error [1]
• \hat\beta provides a point estimate

Frequentist vs. Bayesian Approach

• The Bayesian Approach
  • A regression is constructed using probability distributions, not point estimates as in the frequentist approach
  • Bayes Rule (2) is used to inform the model [1]

p(B|A) = \frac{p(A|B)\cdot p(B)}{p(A)}

• Bayes Rule allows for the calculation of inverse probability (p(B|A) \text{ from } p(A|B))
• p(B|A) \text{ and } p(A|B) are conditional probabilities
• p(A) \text{ and } p(B) are marginal probabilities [2]

Frequentist vs. Bayesian Approach

• The Bayesian Approach
  • Bayesian Inference can be written simply [3]

Posterior = \frac{Likelihood \times Prior}{Normalization}

• The Prior is model of prior knowledge on the subject
• The Likelihood is the probability of the data given the prior
• The Normalization is a constant that ensures the posterior distribution is a valid density function whose integration is equal to 1
• The Posterior is the probability model that expresses an updated view of the model parameters
  • From the initial parameters of the prior
  • Updated with new data expressed in the likelihood function

Frequentist vs. Bayesian Approach

• The Bayesian Approach
  • A more formal expression of Bayes Rule applied for continous parameters

\begin{align*} p(\theta|y) =& \frac{ L(\theta|y)p(\theta) }{p(y)}\\ \\ p(\theta|y) \propto & \text{ }L(\theta|y)p(\theta) \end{align*}

• The normalization constant (p(y) above) ensures the posterior distribution is a valid distribution
  • The posterior density function can be written without this constant
• The resulting prediction is not a point estimate, but a distribution [4]

The Bayesian Approach

• The Bayesian Linear Regression Model
  • changes based on:
    • distribution chosen for regression
    • distribution and hyperparameters chosen for priors
  • Our test case models a continuous outcome by a continuous predictor, so we use a Normal model with conjugate normal priors

Role of prior knowledge in shaping predictions

• Priors can be subjective or objective
  • objective is preferred
• Noninformative priors can be used when there is not adequate prior knowledge
• Discounted priors are the result of adjusting a known prior to better reflect the current data[2]

Figure from [2]

Understanding the Bayesian Framework

Interpreting the Posterior

• The marginal posterior density function (output) may not be available
• Makov Chain Monte Carlo is commonly used
  • Markov chain sequence establishes a sample space from the posterior
  • Integration of samples generated through Monte Carlo techniques from the sample space
• Some popular MCMC algorithms:
  • Gibbs sampler
  • Metropolis-Hastings (MH) [2]

The Model

\begin{align*} Y_i|\beta_0, \beta_1, \sigma &\overset{\text{ind}}{\sim} N (\mu_i, \sigma^2) && \text{with } && \mu_i = \beta_0 + \beta_1X_i \end{align*} Where: - Y_i is the arrival delay for the i-th flight - X_i is the departure delay for the i-th flight - \mu_i = \beta_0 + \beta_1X_i is the local mean arrival delay, , specific to the departure time - \sigma^2 is the variance of the errors - \overset{\text{ind}}{\sim} indicates conditional independence of each arrival delay with the given parameters

Prior Selection

• Regression parameters
  • Intercept: \beta_0 \sim N(m_0, s^2_0)
  • Slope: \beta_1 \sim N(m_1, s^2_1)
  • Error: \sigma \sim \text{Exp}(l)

The Bayesian Linear Regression Model

The model can be written as

\begin{align*} Y_i|\beta_0, \beta_1, \sigma &\overset{\text{ind}}{\sim} N (\mu_i, \sigma^2) && \text{with } && \mu_i = \beta_0 + \beta_1X_i \\ \beta_{0} &\sim N(m_0, s_0^2)\\ \beta_1 &\sim N(m_1, s_1^2)\\ \sigma &\sim \text{Exp}(l) \end{align*}

Tuning Hyperparameters

\begin{align*} \beta_{0c} &\sim N(2, 36^2)\\ \beta_{1} &\sim N(0.02, 0.01^2)\\ \sigma &\sim \text{Exp}(0.02) \end{align*}

The Updated Model

\begin{align*} Y_i|\beta_0, \beta_1, \sigma &\overset{\text{ind}}{\sim} N (\mu_i, \sigma^2) && \text{with } && \mu_i = \beta_0 + \beta_1X_i \\ \beta_{0} &\sim N(2, 36^2)\\ \beta_1 &\sim N(0.02, 0.01^2)\\ \sigma &\sim \text{Exp}(0.02) \end{align*}

Statistical Programming

• Data was analyzed in R [5] , imported via CSV.
  • Data sourced from the Bureau of Transportation Statistics via Kaggle [6].
• Libraries
  • rstanarm [7]
    • stan_glm() function - simulation of model
    • posterior_predict() function - simulation of posterior
  • bayesrules [8]
    • prediction_summary() function - evaluation of posterior

Data Exploration and Visualization

• Describe your data sources and collection process.

• Present initial findings and insights through visualizations.

• Highlight unexpected patterns or anomalies.

Modeling

The Normal Data Model: Departure Time Predictor

\begin{align*} Y_i|\beta_0, \beta_1, \sigma &\overset{\text{ind}}{\sim} N (\mu_i, \sigma^2) && \text{with } && \mu_i = \beta_0 + \beta_1X_i \\ \beta_{0} &\sim N(m_0, s_0^2)\\ \beta_1 &\sim N(m_1, s_1^2)\\ \sigma &\sim \text{Exp}(l) \end{align*}

###The Normal Data Model: Week Day Predictor{.scrollable}

\begin{align*} Y_i|\beta_0, \beta_1, ... \beta_6, \sigma &\overset{\text{ind}}{\sim} N (\mu_i, \sigma^2) && \text{with } && \mu_i = \beta_0 + \beta_1X_{i1} + \beta_2X_{i2} + ... \beta_6X_{i6} \\ \beta_{0} &\sim N(m_0, s_0^2)\\ \beta_1 &\sim N(m_1, s_1^2)\\ \sigma &\sim \text{Exp}(l) \end{align*}

MCMC ACF

:::

Conclusion

• Summarize your key findings.

• Discuss the implications of your results.